7.1. Tracing

Why trace an agent instead of logging the final answer?

One request can cross A2A, a model gateway, several model calls, MCP discovery/invocations, tool code, retrieval, confirmation, and storage. A trace preserves timing/status relationships so an operator can find which boundary failed or consumed the latency budget.

How is ADK telemetry enabled?

agent.agent calls setup_telemetry() during application import. The helper sets privacy defaults, lets ADK configure standard OTel providers, and installs one deduplicated OTel logging handler on the agent logger only when a logs-specific or combined OTLP endpoint exists:

def setup_telemetry() -> None:
    os.environ.setdefault("ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS", "false")
    os.environ.setdefault("OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT", "false")
    maybe_set_otel_providers()
    if _otel_logging_configured():
        _install_agent_log_handler()

With no endpoint, a trace-only configuration, or OTEL_SDK_DISABLED=true, it installs no log bridge. adk run, adk web, evals, and A2A all share the same setup.

How do you point a host agent at the collector?

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_SERVICE_NAME=ops-copilot

Kubernetes supplies http://otel-collector:4318 and resource attributes for namespace/environment. agentgateway uses OTLP gRPC on 4317.

What does the collector do?

pipelines:
  traces:
    receivers: [otlp]
    processors: [memory_limiter, batch]
    exporters: [otlp_http/mlflow, span_metrics]

The collector limits memory, batches spans, exports traces to MLflow, and feeds the spanmetrics connector. It is the routing point; application code does not import an MLflow tracing SDK.

The MLflow image idempotently names built-in experiment id 0 as ops-copilot before starting the server. The collector targets id 0, while evaluation code selects MLFLOW_EXPERIMENT_NAME=ops-copilot; both paths therefore land in the same experiment on a fresh or reused volume.

How do you open the trace UI?

For host mode:

cd infra
docker compose -f observability/compose.yaml up --build -d

Open http://localhost:5000, select the ops-copilot experiment, and inspect a trace after one agent request. For Kubernetes, forward the ClusterIP service:

kubectl -n agentops port-forward svc/mlflow 5000:5000

Do not run both stacks on the same host ports.

Does disabling message capture eliminate privacy risk?

No. It prevents ADK/GenAI instrumentation from intentionally duplicating prompt/response bodies into spans. Request metadata, tool names/arguments, errors, custom logs, session storage, and future instrumentation can still contain sensitive data. PII callbacks redact outbound/inbound/tool content, but raw session ingestion occurs before the outbound callback. Review the generated trace schema and retention/access policy with representative data.

What is the tracing checkpoint?

Issue one read-only request, find its trace in MLflow, and identify A2A/model/tool spans and status/timing. Confirm no prompt/response body appears with default capture variables. Then stop the model backend and verify the error trace remains actionable without exposing the underlying exception to the client.